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An  Information  Organization  Tool  for  Planning 
IN  Air  Traffic  Control 


Planning  is  fundamental  to  the  successful  comple¬ 
tion  of  many  complex,  dynamic  tasks.  For  example, 
Miller,  Copeland,  Heaton,  and  McCloskey  (1998) 
found  planning  to  be  important  for  successful  mili¬ 
tary  operations  (see  also  Pew  &  Mavor,  1998);  Xiao, 
Milgram,  and  Doyle  (1997)  found  it  to  be  an  impor¬ 
tant  aspect  of  anesthesiologists’  preparation  for  sur¬ 
gery.  Although  air  traffic  control  has  been  characterized 
as  largely  involving  shorter-term  tactical  decision 
making  (Durso  &  Gronlund,  1999;  Hutton, 
Olszewski,  Thordsen,  &  Kaempf,  1997),  planning  is 
also  an  important  contributor  (e.g.,  David,  1997; 
Dougherty,  Gronlund,  Durso,  Canning,  &  Mills, 
1999).  Roughly  speaking,  by  tactics  we  mean  deci¬ 
sions  that  relate  to  the  current  moment  and  involve 
the  separation  of  (usually)  pairs  of  aircraft;  planning 
decisions  occur  further  in  advance  than  tactics  and 
involve  the  consideration  of  a  larger  number  of  aircraft. 

An  examination  of  planning  in  air  traffic  control  is 
timely,  given  future  concepts  being  proposed.  For 
instance,  there  have  been  discussions  regarding  the 
creation  of  a  strategic  controller  position  (N.  Lawson 
&  K.  Thompson,  personal  communication,  Dec.  15, 
1 997 ;  see  also  Vivona,  Ballin,  Green,  Bach,  &  McNally, 
1996).  The  proposal  provides  for  one  person  who 
would  be  responsible  for  a  multiple-sector  airspace, 
making  decisions  about  traffic  in  that  airspace,  and 
delegating  responsibility  for  tactical  decisions  to  sec¬ 
tor-level  controllers.  One  goal  of  our  project  was  to 
develop  possible  interface  tools  for  a  strategic  control¬ 
ler  position. 

Before  developing  new  interface  tools  to  aid  plan¬ 
ning  or  to  support  a  strategic  controller  position,  it  is 
important  to  better  understand  how  controllers  use 
their  current  tools  to  develop  plans.  As  pointed  out  by 
David  (1997,  p.  13),  this  is  nontrivial  because  it 
“appears  to  be  a  cognitive  task  almost  invisible  to  the 
outside  observer.  ”  In  addition,  tactics  and  planning 
are  normally  confounded  in  air  traffic  control  because 
both  types  of  decision-making  often  lie  within  the 
same  head,  even  when  a  team  has  responsibility  for  a 
sector  of  airspace. 


Dougherty  et  al.  (1999)  solved  both  problems  by 
assigning  the  role  of  the  planner  and  the  role  of  the 
tactician,  the  implementer  of  the  plan,  to  two  differ¬ 
ent  people  and  requiring  that  the  planner  verbalize  the 
plan  for  the  tactician  to  implement.  The  experiment 
began  with  instructions  that  delineated  the  roles  of 
the  tactician,  the  subject  matter  expert  in  the  experi¬ 
ment,  and  the  planner  who  served  as  the  participant. 
The  tactician’s  job  was  to  maintain  separation  be¬ 
tween  aircraft;  that  is,  he  or  she  made  whatever  alti¬ 
tude,  speed,  and  heading  changes  were  necessary  to 
maintain  separation.  The  planner’s  job  was  to  give  the 
tactician  a  plan  for  managing  the  flow  of  traffic  in  the 
sector  (a  volume  of  airspace).  The  planners  conveyed 
their  plan  to  the  tactician  by  verbalizing  it.  The 
planners  sat  beside  the  radar  (the  tactician  sat  in  front 
of  it)  and  in  front  of  the  flight  progress  strips.^ 

Twelve  en  route  air  traffic  controllers  serving  as 
instructors  at  the  FAA  Academy  in  Oklahoma  City 
completed  two  different  types  of  scenarios.  In  one 
scenario,  the  aircraft  flew  standard  routes  (so-called 
highways  in  the  sky)  in  which  pairs  of  aircraft  would 
only  intersect  at  a  restricted  number  of  points  along 
the  route.  The  primary  problem  to  be  solved  involved 
sequencing  aircraft  into  Dallas/Fort  Worth.  For  the 
other  scenario,  aircraft  flew  direct  routes  through  the 
sector,  which  meant  that  pairs  of  aircraft  could  inter¬ 
sect  at  any  point  in  the  sector.  In  this  scenario,  aircraft 
were  en  route  to  many  different  destinations.  The 
scenarios  were  initially  presented  in  a  paused  mode 
that  allowed  the  planner  the  necessary  time  to  formu¬ 
late  a  plan.  In  a  pilot  study,  we  found  that  the  planner 
would  immediately  fall  into  a  tactical  mode  if  the 
problem  was  active  when  first  viewed. 

The  direct  route  scenario  required  more  decisions 
relating  to  the  separation  of  pairs  of  aircraft  rather 
than  decisions  about  larger  groups  of  aircraft.  How¬ 
ever,  the  latter  were  much  more  frequent  in  the 
sequencing  scenario,  and  because  these  problems  are 
more  the  province  of  a  strategic  controller,  we  focused 
on  sequencing  problems  in  the  interface  development 
effort  to  follow.  In  the  sequencing  scenario,  planners 


‘Flight  progress  strips  for  each  aircraft  were  stacked  vertically  in  a  strip  bay  adjacent  to  the  radar  display.  Flight  strips  are  20  x  3  cm  rectangular 
paper  strips.  They  have  31  fields  of  information,  including  the  call  sign,  aircraft  type,  requested  altitude,  requested  speed,  route  of  flight,  etc. 
The  controllers  mark  on  these  strips  to  update  this  information.  In  the  field,  paper  strips  are  important,  in  part,  because  they  can  function 
as  a  manual  backup  in  case  the  radar  and  computers  fail. 
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made  greater  use  of  the  flight  progress  strips.  Ten  of 
the  12  planners  built  their  sequence  of  aircraft  using 
only  the  strips  (one  planner  used  the  strips  and  the 
radar,  the  other  used  only  the  radar).  In  addition,  a 
greater  proportion  of  the  strips  were  marked  in  the 
sequencing  scenario  (.45  vs.  .24,  all  p"s  <  .05  unless 
otherwise  indicated),  and  more  strips  were  moved  at 
the  beginning  of  the  problem  (5.7  vs.  1.5).  This 
occurred  despite  the  fact  that  the  planners  were  in¬ 
structed  that  the  tactician  would  not  use  the  strips  and 
that  they  were  only  for  the  planner’s  benefit. 

Several  sub-tasks  performed  by  the  controllers  in 
Dougherty  et  al.  (1999)  related  to  their  decision¬ 
making  in  the  sequencing  scenario.  The  two  most 
important  tasks  were  classifying  the  aircraft  into  groups 
and  sequencing  the  aircraft  within  each  group.  The 
classifications  were  needed  to  identify  the  set  of  air¬ 
craft  that  would  be  affected  by  a  particular  constraint 
such  as  “maintain  10  miles  in  trail  between  aircraft  on 
jetroute  107.”  The  sequences  were  needed  to  deter¬ 
mine  in  which  order  aircraft  would  pass  a  common 
ground  point  fix.  These  fixes  were  related  to  the  route 
or  destination  of  the  aircraft  (e.g.,  a  hand-off  location 
to  the  next  sector) .  T o  perform  their  classification  and 
sequencing  sub-tasks,  planners  needed  to:  1)  find 
current  flight  status  information  (position,  altitude, 
and  speed),  and  2)  find  current  flight  plan  informa¬ 
tion  (destination  and  intermediate  points).  All  of  the 
planners  had  grouped  their  flight  progress  strips  by 
aircraft  destination  and  then  ordered  them  within 
each  group  by  arrival  time  at  a  particular  fix.  The 
strips  served  as  tokens  for  the  aircraft,  and  the  stacking 
order  of  the  tokens  made  the  sequence  order  explicit. 

To  enhance  the  functionality  of  the  paper  flight 
strips,  a  software  interface  was  designed  (see  Canning, 
Johansson,  Gronlund,  Dougherty,  &  Mills,  1999). 
Two  screens  were  created  for  the  planner:  a  radar  view 
and  a  flight  organizer  (see  Figures  1  and  2).  The 
planner’s  radar  display  looked  much  like  the  radar 
portion  of  the  Display  System  Replacement  (DSR) 
screens  used  by  the  FAA.  However,  the  radar  display 
was  enhanced  to  allow  for  the  color  coding,  marking, 
and  highlighting  of  the  aircraft  in  coordination  with 
the  flight  organizer  display.  Classifying  and  sequenc¬ 
ing  took  place  on  the  flight  organizer  display.  Each 
aircraft  was  represented  by  a  small  rectangle  called  an 
aircraft  token.  Sets  of  aircraft  tokens  were  placed  into 
containers  called  queuing  blocks  which  represented 
the  groups  of  classified  aircraft.  All  aircraft  tokens 
placed  in  a  queuing  block  were  displayed  in  the  same 
color.  The  corresponding  aircraft  symbol  on  the  radar 
view  appeared  in  that  same  color.  Therefore,  the 
classification  was  encoded  using  color  on  both  dis¬ 


plays.  Planners  created  and  positioned  as  many  queu¬ 
ing  blocks  as  they  wished  on  the  flight  organizer 
screen  and  selected  a  fix  for  each  one.  Time  or  distance 
to  reach  the  fix  (at  the  option  of  the  planner)  was 
displayed  to  the  right  of  each  aircraft  token.  An 
automated  sequencing  tool  allowed  the  aircraft  to¬ 
kens  to  be  automatically  reordered  so  that  the  lowest 
token  was  the  one  closest  to  that  block’s  fix,  and  the 
other  tokens  were  in  ascending  order. 

The  automated  sequencing  tool  did  not  create  an 
optimized  ordering  of  tokens  (as  would  a  planning  aid 
like  CTAS,  Vivona  et  ah,  1996;  or  URET,  Brudnicki 
&  McFarland,  1997).  Rather,  it  ordered  the  tokens  by 
the  time  or  distance  to  reach  the  chosen  fix,  based  on 
the  aircraft’s  current  position,  heading,  and  speed.  It 
provided  a  first  attempt  at  a  sequence  order  that  could 
then  be  fine-tuned  by  the  planner.  This  was  in  keeping 
with  a  philosophy  of  automation  that  keeps  the  opera¬ 
tor  in  the  loop  (human-centered  automation,  e.g., 
Billings,  1996).  According  to  this  philosophy,  new 
tools  should  allow  more  traffic  to  be  handled  and 
better  decisions  to  be  made  but  do  so  not  by 
outsourcing  the  cognitive  ability  of  the  operator  to  the 
tools.  Rather,  through  restructuring  the  information 
and  speeding  access  to  critical  information,  an 
operator’s  cognitive  abilities  can  be  enhanced.  For  a 
similar  philosophy  applied  to  air  traffic  control  see  the 
PHARE  Highly  Interactive  Problem  Solver  devel¬ 
oped  for  Eurocontrol  (Meckoff  &  Gibbs,  1994). 

Moertl  et  ah.  (2000)  reported  the  results  of  an 
experiment  that  evaluated  the  new  interface.  The 
same  12  planners  participated  as  in  the  first  experi¬ 
ment.  They  received  about  two  hours  of  training  on 
the  new  interface  prior  to  participating.  They  com¬ 
pleted  two  sequencing  problems:  one  using  the  flight 
organizer  together  with  the  radar,  and  one  using  the 
strips  and  the  radar  (the  color-coding  functionality  of 
the  radar  was  lost  without  the  flight  organizer).  As 
before,  planners  conveyed  their  plan  to  the  tactician 
by  verbalizing  it.  Planner  performance  (rated  by  the 
planner  and  tactician  and  collapsed  over  the  20- 
minute  scenario)  was  judged  to  be  superior  in  the 
flight  organizer  condition  (6.1  vs.  5.2,  9'point  scale, 
9  being  best),  although  the  advantage  was  only  mar¬ 
ginal  during  the  first  1 0-minutes.  The  flight  organizer 
also  resulted  in  a  lower  workload  over  the  20-minute 
scenario  for  both  the  planner  and  the  tactician  (50.4 
vs.  57.8,  measured  by  the  average  of  the  six  indices  of 
the  NASA  Task  Load  IndeX  that  includes  mental 
demand,  physical  demand,  temporal  demand,  frustra¬ 
tion,  effort,  and  the  assessment  of  performance,  which 
was  reverse  scored  (Hart  &  Staveland,  1998).  Al¬ 
though  participants  had  as  much  time  as  they  needed 
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Figure  1.  The  Radar  View:  One  of  the  two  screens  available  to  the  planner.  See  details  in  the  text  and 
in  Canning  et  al.  (1999). 
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to  develop  their  plans,  a  plan  was  developed  5.4 
minutes  faster,  on  average,  using  the  flight  organizer 
than  using  the  strips,  despite  only  2  hours  of  prior 
familiarization  with  the  interface. 

The  new  interface  seemed  to  offer  several  advan¬ 
tages  over  the  existing  paper  flight  progress  strip 
environment.  The  current  experiment  sought  to  iden¬ 
tify  the  components  of  the  new  interface  that  were 
responsible  for  the  benefit.  In  particular,  the  current 
experiment  evaluated  the  effectiveness  of  the  auto¬ 
mated  sequencing  tool.  In  two  of  the  three  scenarios, 
the  automated  sequencing  tool  was  fully  functional. 
In  the  third  scenario,  the  ability  to  use  fixes  with 
queuing  blocks  was  disabled.  Without  a  fix,  the  auto¬ 
mated  sequencing  tool  was  of  no  use  and  planners 
would  have  to  construct  their  own  sequences  of  air¬ 
craft.  The  planners  could  still  create  queuing  blocks, 
but  they  could  not  reorder  the  tokens  within  a  block 
at  the  touch  of  a  button. 

One  new  feature  was  added  to  the  interface  for  this 
experiment.  The  planner  now  had  the  capability  to 
mark  on  the  aircraft  token  the  control  actions  they 
decided  should  be  taken  to  achieve  the  planned  se¬ 
quence.  Planners  typed  the  appropriate  letter  and  then 
clicked  a  token  to  indicate  that  a  speed,  vector,  and/ 
or  altitude  control  action  was  needed.  The  letters 
appeared  within  the  token  in  the  center  of  the  bottom 
row  (see  Figure  2) .  The  planner  could  mark  more  than 
one  control  action  for  a  particular  aircraft,  and  change 
it  whenever  necessary.  The  planner  could  also  indicate 
that  no  control  action  was  needed  for  a  given  aircraft 
(i.e.,  it  would  fall  into  place  given  its  current  speed, 
altitude,  and  heading).  For  example,  the  token  for 
TWA825  in  Figure  2  shows  that  speed  should  be  used 
by  the  tactician  to  achieve  the  appropriate  sequencing 
of  this  aircraft,  while  no  action  is  indicated  for 
UAL755,  which  meant  that  it  should  fall  into  place 
given  its  current  speed,  heading,  and  altitude.  Also, 
rather  than  requiring  that  the  planner  verbally  convey 
the  plan  to  the  tactician,  the  planner  simply  printed 
out  a  copy  of  the  sequence  order  for  the  tactician  that 
included  the  accompanying  control  actions.  The  tac¬ 
tician  remained  responsible  for  issuing  the  control 
actions  to  the  pilots,  and  for  issuing  other  control 
actions  needed  to  maintain  the  necessary  separation 
between  aircraft. 

Although  the  aforementioned  annotation  of  air¬ 
craft  tokens  facilitated  communication  of  the  plan  to 
the  tactician,  it  also  allowed  for  the  introduction  of  a 
second  independent  variable  into  the  experiment  in¬ 
volving  plan  detail  or  specificity.  Planners  were  re¬ 
quired  to  create  plans  at  one  of  two  different  levels  of 
specificity.  A  partial  sequence  meant  that  the  planner 


could  sequence  as  many  of  the  aircraft  in  a  queuing 
block  as  they  wished.  A  complete  sequence  meant  that 
the  planner  had  to  create  a  more  detailed  plan  by 
sequencing  all  the  aircraft  within  a  queuing  block. 

Planning  aids  like  URET  (Brudnicki  &  McFarland, 
1997)  and  CTAS  (Vivona  et  ah,  1996)  allow  control¬ 
lers  to  make  more  detailed  plans  further  into  the 
future.  On  the  surface,  this  seems  like  a  good  idea.  But 
such  plans  might  be  particularly  prone  to  cognitive 
rigidity  (e.g.,  Taylor,  Finnie,  &  Hoy,  1997).  Cogni¬ 
tive  rigidity,  a  form  of  functional  fixity,  would  occur 
when  a  planner  stuck  with  an  existing  plan  despite 
evidence  (as  judged  by  an  outside  observer)  that  the 
plan  required  modification.  Cognitive  rigidity  can 
also  be  viewed  as  a  form  of  sunk  costs  (Arkes  & 
Blumer,  1985).  More  time  and  effort  devoted  to 
developing  a  plan  might  make  a  planner  more  willing 
to  maintain  an  existing  plan,  despite  evidence  that  it 
should  be  modified.  As  a  result,  planning  perfor¬ 
mance  was  predicted  to  be  worse  when  a  complete 
sequence  was  required. 

Method 

Participants 

Five  en  route  air  traffic  controllers  participated  as 
planners.  All  had  participated  in  our  prior  interface 
experiment,  which  meant  that  they  had  experience 
using  the  flight  organizer.  None  of  the  remaining 
participants  from  the  prior  experiment  were  available, 
and  time  and  resources  were  unavailable  to  train 
additional  participants  on  the  interface.  All  partici¬ 
pants  were  instructors  at  the  FAA  Academy  and  were 
familiar  with  the  AeroCenter  sector  used  in  the  experi¬ 
ment.  All  were  full-performance  level  (FPL)  control¬ 
lers  (i.e.,  certified  to  work  a  sector  independently). 
They  had  been  FPL  controllers  for  an  average  of  20.5 
years.  They  last  worked  in  the  field  3.8  years  previ¬ 
ously,  on  average.  The  tactician,  our  subject  matter 
expert,  was  also  an  FPL  controller. 

Materials 

The  experiment  was  conducted  at  the  Civil  Aero¬ 
space  Medical  Institute  in  Oklahoma  City.  The  plan¬ 
ner  sat  at  one  end  of  a  long  room,  and  the  tactician  and 
two  ghost  pilots  sat  at  the  other  end  (the  ghost  pilots 
on  either  side  of  the  tactician).  The  ghost  pilots 
controlled  the  simulated  aircraft  based  on  the 
tactician's  instructions.  The  planner  and  tactician 
communicated  using  an  intercom  system  that  simu¬ 
lated  a  radio.  The  tactician  and  the  ghost  pilots 
communicated  over  the  tops  of  the  cubicles  that 
separated  them.  The  tactician  had  a  single  screen  on 
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Figure  2.  The  Flight  Organizer:  One  of  the  two  screens  available  to  the  planner.  See  details  in  the  text 
and  in  Canning  et  al.  (1999). 
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which  a  radar  display  was  shown.  The  planner  had  the 
flight  organizer  and  the  radar  display  screens  de¬ 
scribed  above  (see  Figures  1  and  2). 

Four  scenarios  were  developed  by  the  controller 
expert  serving  as  the  tactician.  The  scenarios  were 
judged  to  exceed  the  workload  level  typically  experi¬ 
enced  in  the  field;  there  was  an  average  of  36. 5  aircraft 
to  be  handled  during  the  10-minute  scenario.  The 
primary  problem  to  be  solved  involved  sequencing 
aircraft  into  two  different  destination  airports.  These 
were  called  the  primary  sequences.  Time  constraints 
limited  the  experiment  to  completion  of  only  three  of 
the  scenarios,  although  participants  were  rotated 
through  all  four. 

Procedure 

Each  participant  completed  a  45-minute  training 
session  on  the  use  of  the  flight  organizer  functions. 
This  was  followed  one  or  two  days  later  by  a  45-minute 
practice  session  with  one  of  the  trainers  (graduate  re¬ 
search  assistants  who  were  familiar  with  the  interface). 
The  experimental  session  began  2-3  days  later. 

There  were  three  conditions  in  the  experiment.  In 
one,  the  planners  created  complete  sequences  for  the 
two  primary  sequences  in  the  scenario.  Control  ac¬ 
tions  needed  to  achieve  the  sequence  had  to  be  desig¬ 
nated  for  all  aircraft  in  the  two  primary  sequences.  In 
the  other  two  conditions,  planners  created  partial 
sequences  for  the  two  primary  sequences.  In  other 
words,  it  was  the  planners’  choice  which  aircraft  had 
control  actions  designated  and  which  aircraft  had 
these  decisions  deferred.  The  automated  sequencing 
tool  was  disabled  for  one  of  these  two  partial  sequence 
problems.  This  meant  that  the  planners  had  to  con¬ 
struct  their  sequences  by  manually  determining  dis¬ 
tances.  The  automated  sequencing  tool  was  enabled 
in  the  other  partial  condition  and  in  the  complete 
sequence  conditions.  In  sum,  the  three  conditions 


were:  enabled-partial,  disabled-partial,  and  enabled- 
complete.  See  Table  1  for  an  overview  of  the  require¬ 
ments  of  the  different  conditions.  The  ordering  of  the 
conditions  was  randomized,  as  was  the  assignment  of 
the  four  scenarios  to  the  three  conditions.^ 

The  instructions  began  by  delineating  the  roles  of 
the  tactician  (the  controller  expert)  and  the  planner 
(the  participant).  The  tactician’s  job  was  to  maintain 
separation  between  aircraft;  he  would  make  whatever 
altitude,  speed,  and  heading  changes  were  necessary  to 
maintain  separation.  Only  the  tactician  would 
communicate  with  the  pilots.  The  planner’s  job  was 
to  sequence  the  aircraft  in  the  two  primary  sequences. 
The  planner  was  asked  to  indicate  how  the  sequencing 
should  be  achieved  by  marking  whether  speed,  alti¬ 
tude,  heading  or  no  control  action  was  to  be  used  to 
provide  the  necessary  spacing  and  sequencing.  The 
planner  could  print  a  copy  of  the  plan  for  the  tactician 
anytime  the  planner  wished.  The  print-out  contained 
an  ordered  list  of  the  aircraft  in  the  two  primary 
sequences  and  the  control  actions  designated  by  the 
planner  to  achieve  the  sequences. 

The  experiment  began  with  the  first  scenario  in  the 
paused  mode.  The  planners  then  configured  the  flight 
organizer  to  their  preference.  In  particular,  all  the 
planners  created  at  least  two  queuing  blocks,  one  for 
each  primary  sequence.  Although  the  planners  were 
shown  how  the  interface  worked  during  training,  we 
did  not  require  that  they  use  the  interface  in  a  particu¬ 
lar  way.  Aircraft  going  to  the  appropriate  destinations 
were  then  placed  into  the  designated  queuing  blocks 
and  sequenced  either  manually  or  by  using  the  auto¬ 
mated  sequencing  tool,  depending  on  the  condition. 

After  completing  the  plan,  the  planners  pressed  a 
button  on  the  interface  to  print  two  hardcopies  of  the 
planned  sequence.  One  copy  was  given  to  the  tacti¬ 
cian,  who  used  this  information  to  implement  the 
plan.  The  other  copy  was  given  to  the  planner,  who 


Table  1 

Experimental  Conditions  with  Constraints 


Aircraft  Sequencing 

Control  Action  Marking 

Fix  Status 

Enabled-Partial 

Partial 

Optional 

Enabled 

Disabled-Partial 

Partial 

Optional 

Disabled 

Enabled-Complete 

Complete 

Mandatory 

Enabled 

^  Time  constraints  forced  us  to  drop  the  disabled-complete  condition.  In  a  pilot  study,  plan  preparation  alone  took  over  20  minutes. 
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was  asked  to  circle  aircraft  considered  to  be  in  groups. 
Then,  for  each  circled  group  of  aircraft,  the  planners 
rated  their  confidence  level  that  the  aircraft  in  a  group 
were  in  their  final  sequence  order  (l=very  certain  to 
5=very  uncertain).  Next,  the  planners  rated  their 
confidence  level  that  the  control  actions  marked  were 
sufficient  to  establish  the  sequence  order  (again,  1  =very 
certain  to  5=very  uncertain).  Once  the  planners  com¬ 
pleted  this  process,  the  scenario  was  started  and  the 
tactician  began  implementing  the  plan.  The  planners 
were  instructed  to  modify  their  plan  as  necessary  and 
to  print  out  the  revised  plan  for  the  tactician  whenever 
they  wished. 

The  scenario  was  stopped  after  1 0  minutes,  and  the 
tactician  completed  the  NASA-TLX  (Hart  & 
Staveland,  1998),  which  assessed  mental  demand, 
physical  demand,  temporal  demand,  frustration,  and 
effort  during  the  preceding  10  minutes.  The  NASA- 
TLX  assessment  of  performance  was  excluded  and 
replaced  by  having  the  planner  and  the  tactician 
independently  answer  five  questions  related  to  the 
planner’s  plan:  the  planner’s  involvement  in  the  tac¬ 
tics,  quality  of  the  initial  plan,  quality  of  revisions, 
planning  effectiveness,  and  strategic  awareness.  A  10- 
minute  break  followed.  After  finishing  the  three  sce¬ 
narios,  the  planner  completed  a  questionnaire  that 
collected  biographical  information.  The  participants 
were  then  debriefed  and  released.  An  experimental 
session  took  approximately  2.5  hours. 


Results 

The  tactician’s  TLX  workload  differed  significantly 
across  conditions  {F{2,  3)  =  15.82,y>  =  .026,  see  Table 
2),  according  to  Hotelling’s  (1931)  latest.  Hotelling’s 
7^  test  was  preferred  to  a  one-way  AN OVA  because  it 
does  makes  no  assumptions  about  the  variance/cova- 
riance  matrix  or  assumptions  about  the  sphericity  of 
the  data.  In  addition,  Marcucci  (1986)  showed  that 
the  7^  test  was  a  more  powerful  test  when  epsilon 
departs  appreciably  from  1.0  (the  Greenhouse-Geisser 
epsilon  was  .585).  The  tactician’s  workload  was  sig¬ 
nificantly  less  in  the  ew<a!^/^’^-partial  condition  than  in 
the  disabled-^2sx\2\  condition.  This  comparison  re¬ 
vealed  that  the  tactician’s  workload  was  reduced  when 
the  planner  had  access  to  the  automated  sequencing 
tool. 

The  tactician  and  the  planner  independently  rated 
the  quality  of  the  planner’s  plan.  The  ratings  were 
combined  in  Table  2  and  in  the  following  analysis. 
Although  the  plan  was  judged  best  in  the  enabled- 
partial  condition  and  worst  in  the  disabled-partial 
condition,  consistent  with  the  tactician’s  reduced 
workload  when  the  planner  used  the  automated  se¬ 
quencing  tool,  there  were  no  significant  differences 
across  conditions  (7^(2,  3)  =  1.87>/>  =  .3,  according  to 
Hotelling’s  7^  test).  Although  the  lack  of  a  significant 
difference  could  be  a  power  problem,  note  that  we  had 
enough  power  to  detect  a  change  in  the  tacticians’ 
TLX  ratings. 


Table  2 

Tactician's  Subjective  Workload  and  Rating  of  Plan  Quality  for  the  Three  Experimental 
Conditions  (Standard  Deviations  Given  in  Parentheses). 


Tactician’s  TLX® 

Rating  of  planner’s 
plan  quality‘s 


Enabled-Complete 

Enabled-Partial 

Disabled-Partial 

51.0 

44.9 

63.6 

(7.08) 

(14.27) 

(17.39) 

5.25 

5.55 

4.98 

(0.97) 

(1.35) 

(0.89) 

®  Averaged  across  the  five  workload  subscaies,  performance  excluded.  Scale  ranges  from  0-100  with  100  indicating 
maximum  workload. 

^  Averaged  across  initial  plan  quality,  quality  of  revisions,  strategic  awareness,  and  strategic  planning.  Scale  ranges  from 
1  to  9,  with  9  indicating  superior  planning.  These  data  represent  the  average  of  the  tactician’s  and  the  planners’  ratings. 
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The  lack  of  a  difference  in  plan  quality  was  prob¬ 
ably  because  the  planners  had  as  much  time  as  they 
wished  to  develop  their  plans.  Therefore,  an  indica¬ 
tion  of  the  effectiveness  of  the  automated  tool  might 
not  manifest  itself  in  an  increased  rating  of  plan 
quality  but  instead  in  a  reduced  time  to  develop  the 
plan.  In  other  words,  as  is  often  the  case  in  cognitive 
psychology,  a  difference  in  processing  efficiency  can 
be  observed  in  either  the  accuracy  or  the  speed,  de¬ 
pending  on  the  parameters  of  the  situation.  For  ex¬ 
ample,  a  difference  in  processing  efficiency  between 
students  completing  their  first  year  versus  their  fifth 
year  of  study  of  a  foreign  language  could  result  in  a 
longer  time  for  the  first-year  student  to  translate  a 
passage,  although  they  may  eventually  be  able  to 
translate  it  as  accurately  as  the  fifth-year  student. 
However,  if  we  give  the  first-year  and  the  fifth-year 
students  the  same  amount  of  time,  less  than  what  is 
needed  by  the  first-year  students,  the  time  spent 
translating  would  not  differ  (by  definition),  but  the 
difference  in  processing  efficiency  would  result  in 
reduced  accuracy  for  the  first-year  students. 

Use  of  the  automated  sequencing  tool  did  reduce 
the  time  needed  to  complete  plan  development.  Plan¬ 
ners  created  their  initial  plan  an  average  of  3.75 
minutes  more  quickly  when  the  automated  sequenc¬ 
ing  tool  was  enabled  than  when  it  was  disabled  (en¬ 
abled-partial  54 1  seconds,  1 80.9  vs.  disabled-partial 
766  seconds,  s  =  275.8).  Although  this  difference  was 
of  marginal  statistical  significance  (by  a  one-tailed 
test,  t{4)  =  1.93,/  =  .063),  an  improvement  in  plan 
development  time  of  this  size  is  of  great  practical 
significance  to  the  controller. 

Of  primary  interest  in  assessing  the  impact  of  plan 
specificity  was  the  comparison  of  the  emhled- complete 
to  the  enshled-partial  condition.  This  manipulation 
resulted  in  control  actions  being  indicated  for  94%  of 
the  aircraft  in  the  two  primary  sequences  in  the 
complete  condition  but  only  65.5%  of  the  aircraft  in 
the  two  partial  conditions.  The  tactician's  workload 
was  greater  in  the  enabled-complete  condition  rela¬ 
tive  to  the  enabled-partial  condition,  but  not  signifi¬ 
cantly  so.  Although  plan  quality  was  judged  worse 
when  the  planner  created  a  more  detailed  plan  in  the 
enabled-complete  condition,  as  was  discussed  above, 
the  difference  was  not  significant  during  the  initial  1 0 
minutes  of  the  scenario.  The  support  for  the  disadvan¬ 
tages  of  increased  plan  specificity  was  much  weaker 
than  the  support  for  the  advantages  of  the  automated 
sequencing  tool. 


Small  differences  across  conditions  for  the  remain¬ 
ing  results  made  the  mean  a  sufficient  summary.  The 
planners  circled  an  average  of  5.2  aircraft  as  making 
up  a  group.  This  was  larger  than  the  group  size  of  2.7 
aircraft  reported  by  Means  et  al,  (1988)  in  a  task  that 
involved  circling  recalled  aircraft  on  a  map.  This 
difference  may  reflect  the  larger  groups  of  aircraft 
considered  by  controllers  in  a  planning  role  versus 
their  more  typical  role.  However,  it  may  also  reflect 
differences  in  aircraft  density,  instructions,  questions 
asked,  etc.  Planners  expressed  high  confidence  in  the 
sequence  order  they  presented  to  tacticians  (m  =  1 .68, 
1  to  5  scale,  1=  very  certain).  The  majority  of  the 
control  actions  marked  were  speed  (mean  frequency 
of  m  =  13.3  aircraft)  or  vector  {m  =  9.4).  Altitude 
changes  were  not  important  for  solving  the  sequenc¬ 
ing  problems,  m  =  0.33.  No  control  actions  were 
indicated  for  5.8  aircraft  on  average.  There  was  a 
marginal  positive  correlation  between  the  number  of 
speed  and  vector  control  actions  (r  =  .48,  p  <  .07). 
Apparently,  rather  than  a  preference  for  planners  to 
set-up  their  sequences  using  speed  or  vectors,  the 
positive  correlation  suggested  that  planners  tended  to 
designate  more  of  both  types  of  control  actions.  How¬ 
ever,  a  median  split  (all  three  conditions  collapsed 
together)  on  the  number  of  control  actions  showed 
that  the  tactician  judged  those  planners  who  desig¬ 
nated  fewer  control  actions  to  have  a  better  plan. 
Perhaps  a  planner  with  a  better  “big  picture”  of  the 
sector  knew  the  one  control  action  that  would  solve  a 
problem,  while  the  planner  with  a  poorer  picture 
would  have  to  solve  the  problem  in  pieces  through  a 
series  of  control  action  adjustments. 

Discussion 

The  results  of  the  experiment  show  consistent  and 
beneficial  effects  of  using  the  automated  sequencing 
tool.  The  tactician's  workload  was  decreased.  Al¬ 
though  plan  quality  was  equivalent  across  conditions, 
plans  were  developed  more  rapidly  using  the  auto¬ 
mated  sequencing  tool.  The  results  of  Moertl  et  al. 
(2000)  were  similar.  In  that  experiment,  planners 
developed  plans  using  the  interface  (including  the 
automated  sequencing  tool)  or  the  paper  strips.  Al¬ 
though  plans  were  developed  far  more  rapidly  using 
the  interface,  there  was  no  significant  difference  in 
plan  quality  during  the  initial  10  minutes  of  the 
scenario.  Moertl  et  al.  found  superior  planning  perfor¬ 
mance  in  the  second  1 0  minutes  of  the  scenario  when 
planners  had  to  update  and  adapt  their  plans  to  the 
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changing  traffic  situations.  In  the  present  experiment, 
the  planners  only  completed  10  minutes  per  scenario. 
Perhaps  differences  in  plan  quality  would  have  appeared 
if  the  scenario  was  run  for  10  additional  minutes. 

The  results  of  the  experiment  hint  at  a  detriment  to 
performance  as  a  result  of  making  more  detailed  plans. 
On  the  one  hand,  it  seems  like  a  good  idea  for 
controllers  to  use  tools  like  URET  and  CTAS  to  look 
further  into  the  future.  On  the  other  hand,  having 
more  detailed  plans,  plans  that  a  controller  might  be 
more  invested  in,  could  result  in  greater  cognitive 
rigidity  and  hence  poorer  performance.  Additional 
studies  that  manipulate  the  level  of  detail  of  a  plan  and 
the  degree  of  commitment  to  it  should  be  conducted 
to  verify  that  planning  aids  actually  are  beneficial. 
However,  future  attempts  to  examine  this  construct 
should  consider  other  manipulations  of  plan  specific- 
ity  to  include  dependent  variables  better  targeted  at 
cognitive  rigidity  or,  perhaps,  situation  awareness. 

An  examination  of  en  route  air  traffic  controller 
planning  showed  that  an  information  organization 
tool  could  enhance  the  solving  of  sequencing  prob¬ 
lems.  Prior  research  showed  that  the  flight  organizer 
was  superior  to  the  existing  paper  flight  strip  environ¬ 
ment.  In  the  current  experiment,  a  component  of  the 
flight  organizer,  a  sequencing  tool  that  sorts  aircraft 
tokens  within  a  queuing  block,  was  shown  to  decrease 
the  workload  of  the  tactician  (who  implemented  the 
plan)  and  speed  plan  development.  Dissecting  a  new 
interface  is  an  important  aspect  of  interface  develop¬ 
ment  as  it  can  help  determine  what  features  of  the 
interface  are  beneficial  to  performance  and  what  as¬ 
pects  of  performance  they  enhance.  In  the  present 
experiment,  we  learned  how  the  automated  sequenc¬ 
ing  tool  enhanced  performance.  Surprisingly,  it  was 
the  tactician’s  workload,  rather  than  the  planner’s, 
which  benefited  from  the  use  of  this  tool.  The  tool 
did,  however,  reduce  the  time  needed  by  planners  to 
develop  their  initial  plan.  By  dissecting  an  interface  in 
this  way,  we  can  determine  what  features  of  the 
interface  should  be  maintained  and  what  those  differ¬ 
ent  features  contribute  to  the  overall  performance  of 
the  system. 

The  flight  organizer  interface  we  designed  for  a 
future  strategic  controller  position  represents  an  alter¬ 
native  approach  to  air  traffic  control  modernization. 
We  first  tried  to  understand  how  the  controllers 
function  in  their  current  environment,  and  then  tried 
to  systematically  enhance  that  functioning  through  tar¬ 
geted  tools  that  kept  the  controller  in  the  loop  rather  than 
outsourcing  the  planning  process  to  software. 
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